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Abstract. Generally, universities have complex and large websites, which include a collection of 
many sub-sites related to the different parts of universities (e.g. registration unit, faculties, depart¬ 
ments). Managers of academic institutions and educational websites need to know types of us¬ 
ability problems that could be found on their websites. This would shed the light on possible weak 
aspects on their websites, which need to be improved, in order to reap the advantages of usable 
educational websites. There is a lack of research which provides detailed information regarding the 
types of specific usability problems that could be found on universities’ websites in general, and 
specifically in Jordan. This research employed the heuristic evaluation method to comprehensively 
evaluate the usability of three large public university websites in Jordan (Hashemite University, the 
University of Jordan, and Yarmouk University). The evaluation involves testing all pages related to 
the selected universities’ faculties and their corresponding departments. A list of 34 specific types 
of usability problems, which could be found on a Jordanian university website, was identified. The 
results provide a description regarding the common types of the problems found on the three Jorda¬ 
nian university sites, together with their numbers and locations on the website. 

Keywords: usability, heuristic evaluation, educational websites, Jordan. 


1. Introduction 

Usability is one of the most important characteristics of any user interface; it measures 
how easy the interface is to use (Nielsen, 2003). Usability has been defined as “the extent 
to which a product can be used by specified users to achieve specified goals with effec¬ 
tiveness, efficiency, and satisfaction in a specified context of use” (ISO 9241-11, 1998). 
Usability does not only evaluate the quality of a website, but also provide managers with 
insights into potential problem areas (Agarwal and Venkatesh, 2002). 

Various usability evaluation methods have been developed to evaluate the usability 
of websites. The methods could be categorized into three categories, in terms of how the 
usability problems are identified, for example by evaluators, users, or tools. Evaluator- 
based methods are aimed at finding usability problems that users might encounter while 
interacting with an interface, from the evaluators’ point of view. Heuristic evaluation 
is one of the most frequently evaluator-based usability evaluation methods. It involves 
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having a number of evaluators assess the user interface, and judge whether it conforms 
to a set of usability principles, namely heuristics, (Nielsen and Molich, 1990). The other 
usability evaluation methods, which involve users in the process of identifying usability 
problems, include observations, questionnaires, and interviews. 

With the fast development of the Internet, the increasing use of www as an infor¬ 
mation-seeking, and the increasing number of educational websites, the importance of 
universities’ websites increased. Academic institutions (e.g. universities, colleges) were 
among the early developers of websites to present themselves on the Internet (Astani 
and Elhindi, 2008; Sandvig and Bajwa, 2004; Peterson, 2006). However, the aim of their 
websites differed with time, due to the technological advances and the increasing number 
of Internet users. For example, in early 1990, university websites started as informational 
websites aiming simply to have a presence on the web (Peterson, 2006; Astani and El¬ 
hindi, 2008). Nowadays, academic websites become a vital part of academic institutions, 
and one of their most visible faces (Peterson, 2006). Therefore, the aim of the websites for 
the academic institutions changed. Research indicated that educational websites aim to: 

• Recruit major stakeholders of academic institutions (e.g. prospective students, pro¬ 
spective faculty, alumni, parents) (Astani and Elhindi, 2008; Astani, 2003; Pierce, 
2005). 

• Provide effective services (e.g. email, virtual learning environment), and online 
communication for their other stakeholders. 

• Provide a way to present their image on the Internet (e.g. academic offering, 
programs, services, students resources) (Astani and Elhindi, 2008; Astani, 2003; 
Mentes and Turan, 2012). 

In order to achieve the mentioned aims, and to respond to the new generation of stu¬ 
dents whom grown up with technology, educational websites should consider usability 
while making and improving their websites. Addressing the usability of educational web¬ 
sites could help students to enjoy the learning experience, increase students’ confidence, 
and encourage students to use the website (Lencastre and Chaves, 2008). Unfortunately, 
a university website design is often based on the perceptions of web designers, and/or 
managers in a university instead of students’ needs. Furthermore, many website designers 
have little knowledge of user interface design and usability engineering, therefore, wasting 
users’ time and causing unnecessary traffic on the Internet (Astani and Elhindi, 2008). 

Despite the importance of making educational websites usable, few studies were 
found in the literature that evaluated the usability of such sites, specifically in the context 
of Arabic websites. The studies that were found employed usability methods, including 
heuristic evaluation, to evaluate the usability of educational websites (Astani and El¬ 
hindi, 2008; Noiwan and Norcio, 2000; Pierce, 2005; Kostaras and Xenos, 2006; Toit and 
Bothnia, 2010). However, these studies did not provide details regarding specific types of 
usability problems that could be found on educational websites. 

The aim of this research is to comprehensively investigate the usability of three uni¬ 
versity websites in Jordan, in order to identify common usability problem areas that could 
be found on Jordanian university websites. The specific objectives for the research are: 
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• To use the heuristic evaluation approach to assess to what extent the three selected 
university websites have conformed to usability principles. 

• To identify common usability problems that could be found on each Jordanian 
university website. 

• To generate a list of common usability problem areas that could be found on a 
Jordanian university website. 

This paper is presented as follows. Section two reviews related work. Section three de¬ 
scribes the methods used. Section four outlines the main results. Section five discusses the 
results in light of the literature. Finally section six reviews and presents some conclusions. 


2. Literature Review 

This section provides an overview regarding related work. It is divided into two sections. 
The first concerns earlier research, which evaluated the usability of educational websites. 
The second reviews studies, which shed the light on preferable website features regarding 
usable educational websites from users’ perspectives. 

2.1. Evaluating the Usability of Educational Websites 

This section reviews studies that evaluated the usability of educational sites. These stud¬ 
ies employed various usability methods in the process of identifying usability problems. 
Astani and Elhindi (2008), for example, employed the heuristic evaluation method to 
evaluate the usability of the top 50 colleges and universities. The study was conducted by 
two experts who evaluated and rated the sites (based on Likert scale) against five charac¬ 
teristics: Information content, navigation, usability, customization, download speed and 
security. The authors indicated that the tested websites had usability problems related 
mainly to old content and inappropriate layout. 

Noiwan and Norcio (2000) also evaluated and compared the usability of two Thai 
and two US academic websites, using web usability checklist that aimed to measure the 
usability indexes of the sites. The checklist was categorized into four major sections: 
Finding the information, understanding the information, supporting user tasks, and pre¬ 
senting the information. Each guideline of the checklist was presented as yes/no question. 
The results showed that the sites had several usability problems including: Lack of a site 
map, old content, lack of navigational tools, and inconsistency problems. The results also 
showed that the Thai websites have additional problems, such as: Ineffective internal 
search functions and language problems (e.g. misspelling words). 

Similarly, Kostaras and Xenos (2006) employed the heuristic evaluation method to 
evaluate the usability of the website of the Hellenic Open University using the ten us¬ 
ability heuristics suggested by Nielsen et al. (1994). They identified only 38 usability 
problems on the university website related to: Lack of navigational support links, incon¬ 
sistency problems (e.g. a variation of font sizes was used), errors in the internal search 
function, and inappropriate design of menu. 
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Furthermore, Toit and Bothnia (2010) investigated the usability of the website of an 
academic marketing department in the University of South Africa, using the heuristic 
evaluation method. They mentioned only few examples regarding the usability problems 
that were identified on the website, which related to: Poor navigation, old content, and 
incomplete information regarding the modules of the department. 

Pierce (2005), however, employed heuristic evaluation, and user testing methods to 
evaluate the usability of Harvard University website. Nielsen et al. (1994)’s ten heuris¬ 
tics were used during the heuristic evaluation. The results identified some usability prob¬ 
lems on the site, related mainly to: Lack of navigational tools, inconsistency problems, 
and inappropriate design of the home page. 

Other studies were found in the literature which employed users in the evaluation and 
identification of usability problems on educational websites. Kasli and Aucikurt (2008), 
for example, employed 54 students to investigate 132 websites of tourism departments 
at universities in Turkey. The results showed that most of the sites: Did not present up¬ 
dated information, did not have an internal search function, did not present their content 
in foreign languages, and did not display important information (e.g. academic calendar, 
FAQ, programs). 

Similarly, Christoun et al. (2006) investigated students’ overall satisfaction with an 
academic website, with regard to its technology, usability, aesthetic, and content using an 
online questionnaire. The results showed that only 53.4% of the students agreed that the 
search function of the site was effective, and 66% agreed that it was easy to find informa¬ 
tion. 

Few studies were found in the literature regarding Arabic websites. The study con¬ 
ducted by Mustafa and Al-Zoua’bi (2008) is an example. They employed a questionnaire 
to evaluate the usability of nine Jordanian university websites, by 252 students. The 
results showed that, although the overall usability level of the websites was acceptable, 
there were some weaknesses in some aspects of the sites (e.g. design, interface, perfor¬ 
mance). However, the study didn’t provide details regarding specific types of usability 
problems that users identified on the university sites. 

Similarly, Hasan (2012a) asked 237 students to provide ratings for nine Jordanian 
university websites using usability criteria. The criteria consisted of five categories: 
Navigation, architecture / organization, ease of use and communication, design, and con¬ 
tent. The students were also asked to report qualitatively what they liked, and disliked 
regarding the design of the websites. The results showed that the common weaknesses 
on the websites include: The design of the sites, the fact that most of the sites were in¬ 
consistent, in terms of: Colors, fonts, the Arabic and English language interface, and the 
design of the pages, and the lack of support to Arabic language. 

2.2. Website Features for Usable Educational Websites 

This section reviews various studies that have contributed to the literature by identifying 
the relative importance of website features in the evaluation of usable educational web¬ 
sites from students’ viewpoint. For example, Astani (2003) conducted a study that iden- 
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tified the most, and least important features on a university website from the perspec¬ 
tive of students. The residts showed that the four most important features were: Online 
admission applications, search tools, simple and clear text, and resources. However, the 
four least important features were related to the inclusion of the site to: A site’s purpose 
statement, school history of excellence, major local business and interests, and a link to 
the region’s weather. 

Similarly, Sandvig and Bajwa (2004) investigated the most liked and disliked web¬ 
site features by 48 students on nine university websites. The results showed that the 
most liked features were: Simple layout, useful information, good use of graphics and 
color, simple and comprehensive menu, well organized information, ease to navigate, 
and user-friendly. However, the least liked features were: Outdated links, incomplete 
and outdated information, confusing menus, inability to find information, cluttered and 
disorganized information, slow download time, and small fonts. 

Alternatively, Guilikson et al. (1999) asked 24 students from six faculties to find the 
answers of six questions on a university website. The results showed that the partici¬ 
pants found the answers to only 62 percentages of the questions. Guilikson et al. (1999) 
identified the reasons in their study, behind the inability of students to find information, 
which mainly related to: Unclear organization of the site’s information, lack of a search 
engine, confusing menus, and poor navigational capabilities. 

Regarding Arabic websites, only one study was found, which was conducted by 
Hasan (2012b). She investigated the relative importance of usability criteria in the 
evaluation of the usability of educational websites from the viewpoint of 237 students. 
The criteria consist of 25 website features distributed into five categories: Navigation, 
organization / architecture, ease of use and communications, design, and content. The 
residts showed that the order of the criteria from the most to the least important in the 
evaluation of the usability of educational websites was: Content, navigation, ease of use 
and communications, design, and organization / architecture. The results also showed 
the website features, which students preferred the most for a usable educational website. 
These include: Navigation support, logical structure of a site, quick downloading of web 
pages, aesthetic design, and up-to date information. 

The literature outlines above proved the usefulness of the heuristic evaluation meth¬ 
od regarding its ability to identify various types of usability problems on educational 
websites. They provide important information regarding types of usability problem that 
could be found on educational websites. However, nearly all the studies, which em¬ 
ployed the heuristic evaluation method, used quantitative data based on questionnaires 
to identify usability problems on the tested educational websites from the evaluators’ 
point of view. Therefore, few number of usability problems, with little details, were 
provided. 

The literature also summarizes studies, which employed users in the identification 
of usability problems on educational websites, and other studies which investigated the 
relative importance of website features, from student’s viewpoint, in the evaluation of 
usable educational websites. These studies are a valuable source of usability problems 
that were identified, and perceived by students on various educational websites. 
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3. Methodology 

In order to select three Jordanian university websites, one of the major international univer¬ 
sity ranking websites was used; this was Eduroute (Eduroute, 2011). The three universities, 
which had the highest ranking, as provided by Eduroute for the year 2011, were selected as 
the sample for this research. The universities were: Hashemite University, the University 
of Jordan, and Yarmouk University. These universities are large public universities with 
a variety of disciplines. Hashemite University was established in 1991, the University of 
Jordan was established in 1962, while Yarmouk University was established in 1976. 

In order to evaluate the usability of the selected university websites using the heuris¬ 
tic evaluation method, two documents were developed: Heuristic guidelines, and a list 
of tasks. The heuristic guidelines document includes a set of comprehensive heuristics, 
specific to educational websites, which was developed earlier by Hasan (2012a; 2012b), 
and used as criteria in the evaluation of educational websites in Jordan. The adopted 
heuristics were organized into five major categories. Table 1 displays the categories, and 
subcategories of the adopted heuristics. 

The list of tasks document includes ten tasks, which represent pages students visit 
usually on a university website. These pages represent the findings obtained from an anal¬ 
ysis of a questionnaire, which aimed to investigate the types of pages visited by students 
on a university website. The questionnaire was given to undergraduate students from 
various faculties at one of the universities in Jordan. The total number of students who 
provided usable responses was 237; 149 males, and 88 females. The participant students 
were in general experienced computer, and web users; 91% had more than three years of 
computer experience, and 75% had used the Internet for more than three years. The 237 


Table 1 

The categories and subcategories of the adopted heuristic guidelines 


Category 

Subcategories 

Navigation 

Navigation support; effective internal search; working links; no broken links; 
no orphan pages. 

Architecture/organization 

Logical structure of site; no deep architecture; simple navigation menu. 

Ease of use and 
communication 

Quick downloading of web pages; easy interaction with a website; contact 
us information; foreign language support. 

Design 

Aesthetic design (e.g. the site is attractive, appealing, and has professional 
first impression); appropriate use of images; appropriate choice of fonts 
(e.g. it is recommended to use bold text to highlight important text, and 
capital letters for entire words and sentences, while the use of italics 
are not recommended, also flashing or blinking text should be avoided 
(Oppenheim and Ward, 2006)); appropriate choice of colors (e.g. no more 
than three colors should be used for the body of text, and no more than two 
should be used for the heading text (Ivory, 2003); appropriate page design; 
consistency. 

Content 

Up-to-date information; relevant information; no under-construction pages; 
accurate Information; information about the university; information about 
the colleges; information about the departments. 
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students listed a total of 540 pages. The pages they specified are summarized in Table 2. 
For example, the available courses page was the most frequently visited page listed by 
the students (21.11%). Five pages listed only once or twice, and are counted in Table 2 
under ‘Others’. These pages include: Photo album, annual book, university newspaper, 
university conferences, and library. 

Five evaluators participated in this research; two usability specialists, and three web 
experts. The evaluators were asked to visit all pages included in the list of tasks, and to 
use the heuristic guidelines while evaluating each website. They were also asked to visit 
all pages related to all faculties, and their corresponding departments on each of the stud¬ 
ied universities’ websites. The evaluation was done independently by each evaluator, and 
completed over four months (May 2012 to August 2012). 

The heuristic evaluators’ comments on the compliance of each site to each heuristic 
principle were grouped together for each site, and categorized under the categories and 
sub-categories of the heuristic guidelines. Each heuristic sub-category of each website 
was examined to identify problems with each site. These problems were classified, and 
similar problems were grouped together to identify common areas of usability problems 
on each website. These were examined to identify common areas of usability problems 
across the three websites. Consequently, 34 common areas of usability problems were 
identified, which suggested identifying 34 problem sub-themes. These 34 problem sub¬ 
themes suggested identifying four main problem themes, based on the types of the identi¬ 
fied problems. The four main problem themes related to: Navigation, design, content, and 
ease of use and communication. The list of problem themes and sub-themes is explained 
in the results, and shown in the Appendix. 

The identified problems were categorized into three levels in terms of their location on 
the websites. The first level is ‘university’, and includes pages related directly to the main 
university website, such as: Academic calendar, university services, and pages related to 


Table 2 

Type of pages visited by students on a university website 


Pages Listed by Students 

Frequency* 

Percentage 

Academic calendar 

31 

5.74% 

Academic staff 

29 

5.37% 

Available courses 

114 

21.11% 

Deanship of student affairs- activities 

7 

1.30% 

Departments 

80 

14.81% 

Faculties 

80 

14.81% 

Others 

9 

1.67% 

Registration unit 

79 

14.63% 

Student services 

22 

4.07% 

Study plans 

44 

8.15% 

University announcements / news 

45 

8.33% 

Total 

540 

100.00% 


*Frequency: how often the page was mentioned by the 
participant students in the questionnaires. 
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the registration unit. The second level is ‘faculty’, and includes all pages related to a fac¬ 
ulty sub-site. The third level is ‘department’, and includes pages related to a department 
sub-site. The frequency number of the 34 specific usability problems, with regards to the 
three levels, was described for each of the tested website, as shown in the Appendix. 

In order to determine the level of usability of the three studied university websites, 
and because of the fact that not all the university pages were investigated, a usability 
index was identified in this research, and calculated for the three websites. The usability 
index represents the number of usability problems found on a website, divided by the 
average number of pages investigated on the site. 


4. Results 

This section includes five sub-sections. The first presents a simple general comparison of 
usability of the three Jordanian university websites. The other four sub-sections explain, 
with regards to the four main problem themes, the type and number of specific usability 
problems identified on the sites. 

4.1. Usability of Jordanian University Websites 

The results show that all the tested websites had a large number of usability problems, as 
shown in Table 3. The results show that the website of Hashemite University has the low¬ 
est usability index compared to the other tested websites, indicating that it has the low¬ 
est number of usability problems per investigated pages, while the website of Yarmouk 
University has the highest usability index, indicating that it has the highest number of 
usability problems per investigated pages. 

The usability index for the three tested websites, with regards to the four main us¬ 
ability problem themes, was calculated. Fig. 1 shows that Hashemite University website 
has the lowest usability index, with regard to the design area compared to the other two 
websites, while the University of Jordan website has the highest usability index in this 
area. This indicates that Hashemite University website has the lowest number of design 
problems per investigated pages, compared to the other two websites, while the Univer¬ 
sity of Jordan website has the highest number of design problems. 

Fig. 1 also shows that the University of Jordan website has the lowest usability index, 
with regard to two areas: Navigation and content, indicating that it has the lowest number 


Table 3 

The usability index for the websites 



Hashemite University 

The University of Jordan 

Yarmouk University 

No. of Usability Problems 

4176 

2926 

3399 

No. of Pages Investigated 

1875 

1129 

1187 

Usability Index 

2.23 

2.59 

2.86 
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■ Hashemite 
University 

■ The University 
of Jordan 

■ Yarmouk 
University 


Navigation Design Content Ease of Use and 

Communication 


Fig. 1. The usability index for the three websites, with regard to the four identified usability areas. 


of navigational and content problems, compared to the other two websites. The results, 
however, show that Yarmouk University website has the lowest usability index, with re¬ 
gard to the ease of use, and communication problems, compared to the other websites, 
indicating that this website has the lowest usability problems in this area. 

Fig. 2 shows the percentages of the types of usability problems that were identified 
on the three websites, with regard to the four main usability problem themes. It shows 
that the highest percentage of the usability problems that were identified on each site is 
related to the design area, while the lowest percentage of the usability problems that were 
found on each site is related to the ease of use and communication area. Fig. 2 also shows 
that the second highest percentage of the usability problems identified on the websites of 
Hashemite University and the University of Jordan is related to the content area. How¬ 
ever, the second highest percentage of usability problems identified on the website of 
Yarmouk University is related to the navigation area. 



Fig. 2. The distribution of the types of usability problems identified on the three websites, 
with regards to the four main usability themes. 
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4.2. Navigational Problems 

Seven common navigational problems were identified on the selected Jordanian univer¬ 
sity websites, as shown in Table 4: 

• The first problem is related to weak navigational support. Large numbers of this 
problem were identified on the websites of Hashemite University, and Yarmouk 
University. These concerned mainly with pages on the websites, related specifi¬ 
cally to various faculties and/or departments, which did not have a navigational 
menu, or links to return back to the corresponding faculty and/or department (e.g. 
programs, student curriculum, and guidance plan pages, which related to all the 
departments of Hashemite University). The Appendix shows the frequency of this 
problem on the three websites, with regard to the three levels. 

• The second problem is concerned with misleading links, where large numbers of 
such problem were found on the websites of Hashemite University and Yarmouk 
University. For example, the link related to the name of the chairman (for all the 
departments of Hashemite University) opened a page that was not expected by the 
evaluators; it opened a page that displays an introduction to the department instead 
of information about the chairman of the department. 

• The third problem is related to links that do not open the destination pages. For 
example, the admission and research link, which was located at the home page of 
some faculties of Hashemite University, don’t open the corresponding destination 
pages. They remain on the same page (the home page). 

• The fourth problem is related to links, which when selected, caused the disappear¬ 
ance of the main menu or part of it. For example, selecting the back to top link (at 
the introduction page of Yarmouk University website), caused the disappearance of 
the corresponding department menu. 

• The fifth problem is related to broken links. It was found that the website of Hash¬ 
emite University had the largest number of this problem, compared to the other 
websites, on pages related to its faculties and departments (Appendix). For exam¬ 
ple, the class schedule link, which was located at the home page of all the faculties 
of Hashemite University website was broken. 


Table 4 

Common usability problem sub-themes related to navigation 


Navigational Problems 

Hashemite 

University 

The University 
of Jordan 

Yarmouk 

University 

1. Weak navigational support 

278 

17 

333 

2. Misleading links 

172 

49 

342 

3. Links do not open the destination pages 

46 

20 

75 

4. Links cause disappearance of the menu 

0 

29 

36 

5. Broken links 

529 

208 

21 

6. Orphan pages 

15 

6 

220 

7. Ineffective internal search 

3 

4 

6 

Total no. of navigational problems 

1043 

333 

1033 
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• The sixth problem is related to orphan pages. The results showed that the website 
of Yarmouk University had the largest number of such problem, compared to the 
other websites, on pages related to its departments (Appendix). For example, the 
links related to the names of the faculty members, located at the faculty members 
page for all the departments of Yarmouk University, opened orphan pages. 

• The seventh problem is concerned with ineffective internal search. The results 
showed that all the websites had problems with the internal search functions, re¬ 
lated to the different universities’ sub sites investigated during this research. 


4.3. Design Problems 

18 usability problems were identified on the Jordanian university websites regarding 
design, as shown in Table 5. Those problems concerned four major issues: Inconsistency 
(problems no. 1-9); inappropriate page design (problems no. 10-15); problems with the 
use of images (problems no. 16-17), and problems with the choice of colors (problem 
no. 18). It is worth mentioing that in this research all the inconsistency problems that 
were identified on the sites were categorized under the inconsistency sub-category even- 
though some of them were not related to the design of the sites (e.g. inconsistency in the 
language of the interface and inconsistency in the content). The four major issues related 
the design problems: 


Table 5 

Common usability problem sub-themes related to design 


Design Problems 

Hashemite 

University 

The University Yarmouk 
of Jordan University 

1. Inconsistency in the language of the interface 

140 

138 

109 

2. Inconsistency in the font case (capital and small) 

163 

76 

6 

3. Inconsistency in the font size 

19 

50 

84 

4. Inconsistency in the font style (regular and bold) 

59 

18 

76 

5. Inconsistency in the content 

10 

28 

15 

6. Inconsistency in the alignment of the header 

2 

19 

4 

7. Inconsistency in the font color 

13 

31 

0 

8. Inconsistency in the design 

12 

0 

0 

9. Inconsistency in the size of images 

0 

0 

52 

10. Inappropriate orientation of the page design 

60 

0 

179 

11. Inappropriate design of main menu 

39 

101 

113 

12. Ineffective text format 

925 

457 

427 

13. Inappropriate/ not representative heading of a page 

95 

432 

142 

14. Long pages 

2 

5 

149 

15. Short pages 

0 

0 

29 

16. Inappropriate quality of images 

57 

19 

29 

17. Broken images 

30 

532 

2 

18. Inappropriate choice of color 

57 

28 

0 

Total no. of design problems 

1683 

1934 

1416 
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• Nine specific inconsistency problems were identified on the websites (Table 5, 
problems no. 1-9). The results revealed that the large number of the inconsis¬ 
tency problems that was found on the sites related to inconsistency in the language 
of the interface. This is related to links at the English interface of a site, which 
opened pages that displayed Arabic content, and vice versa. Other examples of 
inconsistency problems are shown in Table 5. 

• Six problems were found on the sites concerned inappropriate design of their pag¬ 
es (Table 5, problems no. 10-15). One of theses problems related to ineffective 
text format of the sites’ pages, where information, figures, and tables were not 
aligned correctly. The results showed that all the websites had large number of 
such problems (Table 5, Appendix). Another problem, which was commonly iden¬ 
tified on the websites of Hashemite University and Yarmouk University, related 
to inappropriate orientation of the page design. For example, it was found that the 
main menu and the vertical scrollbar, for all the faculties at the English language 
interface of Yarmouk University website, were located at the right side of the page 
instead of the left side. Table 5 also shows that the University of Jordan website 
had many pages without headings or with inappropriate headings. 

• Another two specific types of problems were found on the sites related to inap¬ 
propriate use of images: Poor quality of images, and broken images (Table 5, 
problems no. 16-17). Table 5 shows that the University of Jordan website had the 
largest number of broken images, compared to the other sites, on pages related to 
its faculties and departments (Appendix). For example, message from the dean 
and photo gallery pages for most of the faculties of the University of Jordan web¬ 
site had broken images. 

• The last problem is related to inappropriate choice of color, and was identified on 
Hashemite University, and the University of Jordan websites (Table 5, problem 
no. 18). These websites have pages that used inappropriate combination of back¬ 
ground and font colors. 


4.4. Content Problems 

Seven common usability problems were identified on the Jordanian university websites 
regarding content, as shown in Table 6: 

• The first problem is related to old information, which was presented on pages 
related to various faculties and departments of Hashemite University and the Uni¬ 
versity of Jordan websites (Appendix). Examples on these pages: Latest news and 
activities on Hashemite University website. 

• The second problem is related to incomplete information. The results show that 
all the websites had large number of such problem (e.g. faculty staff pages on 
Hashemite University). 

• Regarding the third problem, the results show that the websites of Hashemite Uni¬ 
versity and Yarmouk University had pages with links that open DOC and/or PDF 
files, but there is no information regarding the type of file a link opened. 
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Table 6 

Common usability problem sub-themes related to content 


Content Problems 

Hashemite 

The University 

Yarmouk 


University 

of Jordan 

University 

1. Old information 

68 

41 

0 

2. Incomplete information 

344 

158 

463 

3. No information regarding the type 
of a file that a link will open 

59 

0 

227 

4. Empty page 

173 

242 

148 

5. Inappropriate content 

444 

80 

62 

6. Punctuation errors 

272 

39 

13 

7. Grammatical accuracy problems 

18 

11 

2 

Total no. of content problems 

1378 

571 

915 


• The fourth problem is related to empty pages, where the University of Jordan web¬ 
site had the largest number of this problem, compared to the other two websites. 

• With regard to the fifth problem, the results showed that all the sites had inappro¬ 
priate content problems. Examples on these problems: Crowded pages, repetition 
of content, and very concise content. 

• Finally, the results identified two specific types of problems related to grammatical 
accuracy: Punctuation and grammatical errors (the sixth and seventh problems at 
Table 6, respectively). An example on pages with spelling errors, which related to 
the University of Jordan website, is the vision and mission page, on the Faculty of 
Arts and design, where there was a spelling error in its header; vission was written 
instead of vision. 


4.5. Ease of Use and Communication Problems 

Two usability problems were identified on the Jordanian university websites regarding 
ease of use and communication, as shown in Table 7: 

• The first problem is related to being not easy to interact with the websites, in order 
to visit some pages. For example, it was not easy to visit the Occupational Therapy 
Department from the corresponding faculty website (Allied Flealth Science related 


Table 7 

Common usability problem sub-themes related to ease of use and communication 


Ease of Use and Communication Problems 

Hashemite 

The University Yarmouk 


University 

of Jordan 

University 

Difficult interaction with a website 

8 

3 

35 

Not supporting more than one language 

64 

85 

0 

Total no. of ease of use and communication problems 

72 

88 

35 
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to Hashemite University), since the faculty main menu did not have an item for it. 
The only way to visit the department was from the university main menu. 

• The second problem is related to the lack of support to the Arabic language. The re¬ 
sults show that the websites of Hashemite University and the University of Jordan 
had problems related to the fact that they did not support the Arabic language. The 
language interface of the Hashemite University website, including its 13 faculties 
and their corresponding departments, was written only in English language. Re¬ 
garding the University of Jordan website, it was found that most of its faculties (16 
out of 18) and their corresponding departments were presented using only English 
language. However, Yarmouk University website presents the university faculties 
and their corresponding departments using Arabic and English languages. 


5. Discussion 

This research addressed a gap noted in the literature, with regard to lack of identification 
of specific types of usability problems found by heuristic evaluation methods on educa¬ 
tional websites, specifically in Jordan. Earlier research, which evaluated the usability of 
educational websites, provided only few examples of the usability problems that were 
identified by the heuristic evaluation method (Astani and Elhindi, 2008; Noiwan and Nor- 
cio, 2000; Pierce, 2005; Kostaras and Xenos, 2006; Toit and Bothnia, 2010). This could 
relate to the fact that in these studies the usability problems that were identified by the 
heuristic evaluators were only based on the quantitative data obtained from various meth¬ 
ods used in their research. This research, however, uniquely identified large number of 
usability problems, related to 34 specific types of usability problems that could be found 
on an educational website. It also presented the frequency number of these problems on 
each of the tested website (Appendix). This is related to the fact that this research used 
the qualitative data obtained from the heuristic evaluators, who comprehensively inves¬ 
tigated large number of pages on the three selected Jordanian university websites, in the 
process of identifying the usability problems. 

Despite the fact that the results of this research involved providing more detailed de¬ 
scriptions of usability problems that were identified by the heuristic evaluation compared 
to the previous research, it was found that there was an agreement between the results 
of this research, and the results of the previous studies. Table 8 summarizes the usability 
problems, which were identified by the heuristic evaluators on educational websites, by 
earlier research. 

These were also confirmed by the results of this research. Specific examples of prob¬ 
lems identified in this research were discussed in Section four. The Appendix provides 
more information regarding the frequency of similar problems on the tested websites, and 
provides information regarding additional problems that were uniquely identified in this 
research. These included: Misleading links, links don’t open the destination pages, links 
cause disappearance of the menu, broken links, various types of inconsistency problems, 
inappropriate orientation of the page design, ineffective text format, broken images, inap¬ 
propriate choice of color, empty page, inappropriate content, difficult interaction with a 
website, and lack of support to the Arabic language. 
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Table 8 

Examples of usability problems identified by the heuristic evaluation method 


Example of Usability Problems 

References 

Old content 

Astani and Elhindi (2008); Noiwan and Norcio (2000); Toit and Bothma 
(2010) 

Lack of navigational support 

Noiwan and Norcio (2000); Kostaras and Xenos (2006); Toit and Bothma 
(2010); Pierce (2005) 

Inconsistency problems 

Noiwan and Norcio (2000); Kostaras and Xenos (2006); Pierce (2005) 

Ineffective internal search 

Noiwan and Norcio (2000); Kostaras and Xenos (2006) 

Language problems 

Noiwan and Norcio (2000) 

Inappropriate page design 

Astani and Elhindi (2008); Pierce (2005) 

Incomplete information 

Toit and Bothma (2010) 

Inappropriate design of menus 

Kostaras and Xenos (2006) 


The large number of common and specific usability problems that were uniquely iden¬ 
tified in this research on the Jordanian universities’ websites, could be used as guidelines 
for universities in Jordan, to investigate and improve their universities’ websites, and 
therefore to achieve the advantages of usable educational websites. 

This research, however, had two limitations. The first related to the fact that it em¬ 
ployed only heuristic evaluators who identified large number of usability problems on the 
three universities’ websites from the viewpoint of evaluators, and not users. The second 
related to the fact that the classification step of usability problems according to their se¬ 
verity (e.g. major and minor problems), from the point of view of the evaluators, could 
not be accomplished. This is related to the lack of evaluators’ time to accomplish this 
task. Because of these limitations, a further step was taken. It aimed at shedding the light 
on some usability problems, among the 34 specific types of problems, which might be 
important from students’ perspective. This was done by making a comparison between 
the 34 specific types of usability problems identified in this research, with the usability 
problems identified by users in earlier research (Section 2). Table 9 shows 17 (out of 
34) types of usability problems that were identified in this research, which were also 
identified by users in earlier studies. These problems could be considered as important 
problems, and could be given greater weight, while improving the usability of Jordanian 
universities’ websites. 


6. Conclusions 

This research comprehensively evaluated the usability of three large public university 
websites in Jordan, namely: Hashemite University, the University of Jordan, and Yar- 
mouk University, including all pages related to their faculties and departments, using the 
heuristic evaluation method. The results identified a list of 34 specific types of common 
usability problems that were found on the selected Jordanian university websites, and 
described the frequency number of these problems, and their locations on the websites. 
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Table 9 

Usability problems identified in this research and usability problems identified by users in earlier research 


Usability problems identified in this research 

Usability Problems identified by users 

Weak navigation support 

Poor navigational capabilities (Guilikson et al., 1999) 

Ineffective Internal search 

Ineffective search function (Christoun et al ., 2006) 

Inconsistency in the language of the interface 
Inconsistency in font case (capital and small) 
Inconsistency in the font size 

Inconsistency in the font style (regular and bold) 
Inconsistency in the content 

Inconsistency in the alignment of the header 
Inconsistency in the font color 

Inconsistency in the design 

Inconsistency in the size of images 

Inconsistency (colors, fonts, Arabic and English 
language interface, design of the pages) (Hasan, 
2012a) 

Inappropriate design of main menu 

Confusing menus (Sandvig and Bajwa, 2004; Guilikson 
etal., 1999) 

Old information 

Old content (Kasli and Aucikurt, 2008; Sandvig and 
Bajwa, 2004; Hasan, 2012a) 

Incomplete information 

Missing information (Kasli and Aucikurt, 2008; 

Sandvig and Bajwa, 2004) 

Inappropriate content 

Cluttered and disorganized information (Sandvig and 
Bajwa, 2004) 

Difficult interaction with a website 

Not easy to find information (Christoun et al., 2006; 
Sandvig and Bajwa, 2004) 

Not supporting more than one language 

Lack of support for the Arabic language (Hasan, 2012a) 


The results of this research illustrated the effectiveness of the heuristic evaluation 
method in identifying a large number of usability problems that could be used to improve 
different aspects of universities’ websites. The large number of usability problems, identi¬ 
fied by the evaluators, could be also identified using software tools (e.g. HTML Toolbox), 
which automatically assess whether a website conform to a set of specific guidelines. The 
software tools, however, assess the quality of the HTML code of a website with regards 
to a number of guidelines, which focus on the accessibility of a site rather than its general 
usability. Furthermore, the results provided by such tools do not provide sufficient details 
regarding types of problems on a website. Unlike software tools, the heuristic evaluation 
method provided details regarding specific types of usability problems that were found by 
the evaluators on the websites. 

The results of this research emphasized the limitations of the heuristic evaluation, 
with regard to the fact that the heuristic evaluators could not play the role of real users, 
and could not predict actual problems that users might face while interacting with the 
sites. Therefore, a further step was taken. This is related to comparing the 34 usability 
problems with earlier studies that identified usability problems on educational sites from 
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the point view of students. The results shed the light on some usability problems (17 out 
of 34), which might be major for students. These problems related to: Lack of navigation¬ 
al support, ineffective internal search, inconsistency problems, inappropriate design of 
main menu, old infonnation, incomplete information, difficult interaction with a website, 
and lack of support to the Arabic language. 

The results are particularly useful for managers, designers, and/or evaluators of uni¬ 
versities’ websites in Jordan, who are willing to evaluate and improve the design of their 
websites in order to reap the advantages of usable educational websites. The 34 specific 
types of common usability problems that were identified in this research provide guid¬ 
ance for them regarding website features, which should be taken into consideration while 
designing and/or evaluating educational websites. 
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Appendix 


Usability problem themes and sub-theme identified in this research, together with the number 
of each problem on each of the tested website with regard to the three levels. 


Hashemite University The University of Yarmouk University 
Jordan 


Navigational Problems 


1. Weak navigational support 

Freq. 




Freq. 




Freq. 





U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


2 

130 

146 

278 

4 

10 

3 

17 

85 

42 

206 

333 

2. Misleading links 

Freq. 




Freq. 




Freq. 





U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


28 

56 

88 

172 

4 

28 

17 

49 

74 

29 

239 

342 

3. Links do not open the destina¬ 

Freq. 




Freq. 




Freq. 




tion pages 

U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


2 

44 

0 

46 


20 

0 

20 

1 

16 

58 

75 

4. Links cause disappearance of 





Freq. 




Freq. 




the menu 



0 


U 

F 

D 

T 

U 

F 

D 

T 






4 

22 

3 

29 

2 


34 

36 

5. Broken links 

Freq. 




Freq. 




Freq. 





U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


9 

274 

246 

529 

3 

74 

131 

208 

12 

3 

6 

21 

6. Orphan Pages 

Freq. 




Freq. 




Freq. 





U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


2 

13 

0 

15 

2 

4 

0 

6 

0 

2 

218 

220 

7. Ineffective internal search 

Freq. 




Freq. 




Freq. 





U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


1 

1 

1 

3 

2 

1 

1 

4 

2 

2 

2 

6 

Total number of navigational 

Freq. 




Freq. 




Freq. 




problems 

U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 


44 

518 

481 

1043 

19 

159 

155 

333 

176 

94 

763 

1033 
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Design Problems 

8. Inconsistency in the lan¬ 
guage of the interface 

9. Inconsistency in font case 
(capital and small) 

10. Inconsistency in the font size 


11. Inconsistency in the font 
style (regular and bold) 

12. Inconsistency in the content 


13. Inconsistency in the align¬ 
ment of the header 


14. Inconsistency in the font 
color 


15. Inconsistency in the design 


16. Inconsistency in the size of 
images 

17. Inappropriate orientation of 
the page design 

18. Inappropriate design of main 
menu 


19. Ineffective text format 


20. Inappropriate/ not representa¬ 
tive heading of a page 


Hashemite University The University of Yarmouk University 
Jordan 


Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

13 

45 

82 

140 

7 

75 

56 

138 

37 

1 

71 

109 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

1 

19 

143 

163 

0 

20 

56 

76 

2 

1 

3 

6 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

11 

8 

19 

0 

15 

35 

50 

3 

27 

54 

84 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

12 

47 

59 

0 

7 

11 

18 

3 

23 

50 

76 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

4 

6 

10 

1 

17 

10 

28 

4 

2 

9 

15 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

2 

0 

2 

0 

4 

15 

19 

0 

4 

0 

4 

Freq. 




Freq. 








U 

F 

D 

T 

U 

F 

D 

T 



0 


0 

8 

5 

13 

0 

11 

20 

31 





Freq. 












U 

F 

D 

T 



0 




0 


0 

12 

0 

12 





Freq. 






0 




0 


U 

F 

D 

T 









0 

52 

0 

52 

Freq. 








Freq. 




U 

F 

D 

T 



0 


U 

F 

D 

T 

1 

15 

44 

60 





7 

29 

143 

179 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

23 

16 

39 

0 

41 

60 

101 

4 

18 

91 

113 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

31 

263 

631 

925 

0 

110 

347 

457 

34 

51 

342 

427 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

3 

86 

6 

95 

2 

145 

285 

432 

5 

26 

111 

142 

Freq. 




Freq. 




Freq. 




U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 

T 

0 

2 

0 

2 

1 

3 

1 

5 

4 

19 

126 

149 


21. Long pages 
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22. Short pages 

23. Inappropriate quality of 
images 

24. Broken images 

25. Inappropriate choice of color 

Total number of design 
problems 

Content Problems 

26. Old information 

27. Incomplete information 

28. No information regarding the 
type of a file a link will open 

29. Empty page 

30. Inappropriate content 

31. Punctuation errors 

32. Grammatical accuracy 
problems 

Total number of content 
poblems 


Hashemite University The University of Yarmouk University 
Jordan 










Freq. 






0 
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D 

T 
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26 

2 

29 
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Freq. 
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T 
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28 
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19 
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29 
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29 
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U 

F 

D 

T 

U 

F 

D 

T 

U 

F 

D 
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19 

6 

30 
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58 
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1 

0 

1 

2 
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U 
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T 

U 

F 

D 

T 



0 


0 

13 

44 

57 
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T 
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26 

68 

1 
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15 
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Hashemite University The University of Yarmouk University 
Jordan 


Ease of Use and Communication Problems 


33. Difficult interaction with a 

Freq. 
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34. Not supporting more than 
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Total number of ease of use 
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and communication problems 
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D 
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13 

13 

46 
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3 
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71 
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Freq.: Frequency of the problem D: Department level 

U: University level T: Total no. of problems for the problem sub-theme 

F: Faculty level 
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Trijij Jordanijos universitetq svetainiq euristinis vertinimas 

Layla HASAN 

Dazniausiai universitetai turi dideles kompleksines svetaines, apimancias kitas svetaines susie- 
tas su skirtingomis universiteto atliekamomis funkcijomis (pvz., registracijos svetaine, fakultetq, 
katedrq svetaines). Akademinii} instituciji} ir edukaciniq svetainiq vadovai turi zinoti iskylanciq 
svetainii} naudojimo problemij grapes. Problemij isaiskinimas turetq padeti surasti svetainiq tru- 
kumus, kuriuos pasalinus butq pasiektas efektyvesnis edukaciniq svetainiq naudojimas. Pastebeta 
tyrimq, skirtq svetainiq naudojimo problemoms, tipq analizes stoka. Tyrime taikomas euristinis 
vertinimo metodas, kuriuo siekiama jvertinti trijq didziqjq Jordanijos universitetq (Hashemite, Jor¬ 
danijos ir Yarmouk universiteto) svetainiq naudojimq. Vertinimas apima visq tinklalapiq, susietq su 
atrinktais universitetq fakultetais ir atitinkamomis katedromis testavimq. Nustatyti 34 Jordanijos 
universitetq svetainese rasti jq naudojimo problemq tipai. Pateikiami rastq tipq aprasai su konkre- 
ciais pavyzdziais. 





